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Abstract 

Source and channel coding over multiuser channels in which receivers have access to correlated 



o 
o 

source side information is considered. For several multiuser channel models necessary and sufficient 



conditions for optimal separation of the source and channel codes are obtained. In particular, the multiple 
access channel, the compound multiple access channel, the interference channel and the two-way channel 
with correlated sources and correlated receiver side information are considered, and the optimality of 
separation is shown to hold for certain source and side information structures. Interestingly, the optimal 



rya ' separate source and channel codes identified for these models are not necessarily the optimal codes for 

the underlying source coding or the channel coding problems. In other words, while separation of the 

source and channel codes is optimal, the nature of these optimal codes is impacted by the joint design 
> 
\Q ' criterion. 

^i '. I. Introduction 

Shannon's source-channel separation theorem states that, in point-to-point communication 
systems, a source can be reliably transmitted over a channel if and only if the minimum source 
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coding rate is below the channel capacity [1]. This means that a simple comparison of the rates 
of the optimal source and channel codes for the underlying source and channel distributions, 
respectively, suffices to conclude whether reliable transmission is possible or not. Furthermore, 
the separation theorem dictates that the source and channel codes can be designed independently 
without loss of optimality. This theoretical optimality of modularity has reinforced the notion 
of network layers, leading to the separate development of source and channel coding aspects 
of a communication system. The separation theorem holds for stationary and ergodic sources 
and channels under the usual information theoretic assumptions of infinite delay and complexity 
(see [2] for more general conditions under which separation holds). However, Shannon's source- 
channel separation theorem does not generalize to multiuser networks. 

Suboptimality of separation for multiuser systems was first shown by Shannon in [3], where 
an example of correlated source transmission over the two-way channel was provided. Later, a 
similar observation was made for transmitting correlated sources over multiple access channels 
(MACs) in [4]. The example provided in [4] reveals that comparison of the Slepian-Wolf source 
coding region [5] with the capacity region of the underlying MAC is not sufficient to decide 
whether reliable transmission can be realized. 

In general communication networks have multiple sources available at the network nodes, 
where the source data must be transmitted to its destination in a lossless or lossy fashion. Some 
(potentially all) of the nodes can transmit while some (potentially all) of the nodes can receive 
noisy observations of the transmitted signals. The communication channel is characterized by 
a probability transition matrix from the inputs of the transmitting terminals to the outputs of 
the receiving terminals. We assume that all the transmissions share a common communications 
medium; special cases such as orthogonal transmission can be specified through the channel 
transition matrix. The sources come from an arbitrary joint distribution, that is, they might be 
correlated. For this general model, the problem we address is to determine whether the sources 
can be transmitted losslessly or within the required fidelity to their destinations for a given 
number of channel uses per source sample (cupss), which is defined to be the source-channel 
rate of the joint source channel code. Equivalently, we might want to find the minimum source- 
channel rate that can be achieved either reliably (for lossless reconstruction) or with the required 
reconstruction fidelity (for lossy reconstruction). 

The problem of jointly optimizing source coding along with the multiuser channel coding 
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in this very general setting is extremely complicated. If the channels are assumed to be noise- 
free finite capacity links, the problem reduces to a multiterminal source coding problem [1]; 
alternatively, if the sources are independent, then we must find the capacity region of a general 
communication network. Furthermore, considering that we do not have a separation result for 
source and channel coding even in the case of very simple networks, the hope for solving this 
problem in the general setting is slight. 

Given the difficulty of obtaining a general solution for arbitrary networks, our goal here 
is to analyze in detail simple, yet fundamental, building blocks of a larger network, such as 
the multiple access channel, the broadcast channel, the interference channel and the two-way 
channel. Our focus in this work is on lossless transmission and our goal is to characterize the 
set of achievable source-channel rates for these canonical networks. Four fundamental questions 
that need to be addressed for each model can be stated as follows: 

1) Is it possible to characterize the optimal source-channel rate of the network (i.e., the mini- 
mum number of channel uses per source sample (cupss) required for lossless transmission) 
in a computable way? 

2) Is it possible to achieve the optimum source-channel rate by statistically independent source 
and channel codes? By statistical independent source and channel codes, we mean that the 
source and the channel codes are designed solely based on the distributions of the source 
and the channel distributions, respectively. In general, these codes need not be the optimal 
codes for the underlying sources or the channel. 

3) Can we determine the optimal source-channel rate by simply comparing the source coding 
rate region with the capacity region? 

4) If the comparison of these canonical regions is not sufficient to obtain the optimal source- 
channel rate, can we identify alternative finite dimensional source and channel rate regions 
pertaining to the source and channel distributions, respectively, whose comparison provides 
us the necessary and sufficient conditions for the achievability of a source-channel rate? 

If the answer to question (3) is affirmative for a given setup, this would maintain the optimality 
of the layered approach described earlier, and would correspond to the multiuser version of Shan- 
non's source-channel separation theorem. However, even when this classical layered approach 
is suboptimal, we can still obtain modularity in the system design, if the answer to question 
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(2) is affirmative, in which case the optimal source-channel rate can be achieved by statistically 
independent source and channel codes, without taking the joint distribution into account. 

In the point-to-point setting, the answer to question (3) is affirmative, that is, the minimum 
source-channel rate is simply the ratio of the source entropy to the channel capacity; hence 
these two numbers are all we need to identify the necessary and sufficient conditions for the 
achievability of a source-channel rate. Therefore, a source code that meets the entropy bound 
when used with a capacity achieving channel code results in the best source-channel rate. In 
multiuser scenarios, we need to compare more than two numbers. In classical Shannon separation, 
it is required that the intersection of the source coding rate region for the given sources and the 
capacity region of the underlying multiuser channel is not empty. This would definitely lead to 
modular source and channel code design without sacrificing optimality. However, we show in 
this work that, in various multiuser scenarios, even if this is not the case for the canonical source 
coding rate region and the capacity region, it might still be possible to identify alternative finite 
dimensional rate regions for the sources and the channel, respectively, such that comparison 
of these rate regions provide the necessary and sufficient conditions for the achievability of a 
source-channel rate. Hence, the answer to question (4) can be affirmative even if the answer to 
question (3) is negative. Furthermore, we show that in those cases we also have an affirmative 
answer to question (2), that is, statistically independent source and channel codes are optimal. 

Following [6], we will use the following definitions to differentiate between the two types of 
source-channel separation. Informational separation refers to classical separation in the Shannon 
sense, in which concatenating optimal source and channel codes for the underlying source 
and channel distributions result in the optimal source-channel coding rate. Equivalently, in 
informational separation, comparison of the underlying source coding rate region and the channel 
capacity region is sufficient to find the optimal source-channel rate and the answer to question 

(3) is affirmative. Operational separation, on the other hand, refers to statistically independent 
source and channel codes that are not necessarily the optimal codes for the underlying source 
or the channel. Optimality of operational separation allows the comparison of more general 
source and channel rate regions to provide necessary and sufficient conditions for achievability 
of a source-channel rate, which suggests an affirmative answer to question (4). These source 
and channel rate regions are required to be dependent solely on the source and the channel 
distributions, respectively; however, these regions need not be the canonical source coding 
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rate region or the channel capacity region. Hence, the source and channel codes that achieve 
different points of these two regions will be statistically independent, providing an affirmative 
answer to question (2), while individually they may not be the optimal source or channel codes 
for the underlying source compression and channel coding problems. Note that the class of 
codes satisfying operational separation is larger than that satisfying informational separation. We 
should remark here that we are not providing precise mathematical definitions for operational 
and information separation. Our goal is to point out the limitations of the classical separation 
approach based on the direct comparison of source coding and channel capacity regions. 

This paper provides answers to the four fundamental questions about source-channel coding 
posed above for some special multiuser networks and source structures. In particular, we consider 
correlated sources available at multiple transmitters communicating with receivers that have 
correlated side information. Our contributions can be summarized as follows. 

• In a multiple access channel we show that informational separation holds if the sources 
are independent given the receiver side information. This is different from the previous 
separation results [7]- [9] in that we show the optimality of separation for an arbitrary 
multiple access channel under a special source structure. We also prove that the optimality 
of informational separation continue to hold for independent sources in the presence of 
correlated side information at the receiver, given which the sources are correlated. 

• We characterize an achievable source-channel rate for compound multiple access channels 
with side information, which is shown to be optimal for some special scenarios. In particular, 
optimality holds either when each user's source is independent from the other source and 
one of the side information sequences, or when there is no multiple access interference at 
the receivers. For these cases we argue that operational separation is optimal. We further 
show the optimality of informational separation when the two sources are independent given 
the side information common to both receivers. Note that the compound multiple access 
channel model combines both the multiple access channels with correlated sources and the 
broadcast channels with correlated side information at the receivers. 

• For an interference channel with correlated side information, we first define the strong 
source-channel interference conditions, which provide a generalization of the usual strong 
interference conditions [10]. Our results show the optimality of operational separation under 
strong source-channel interference conditions for certain source structures. 
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• We consider a two-way channel with correlated sources. The achievable scheme for com- 
pound MAC can also be used as an achievable coding scheme in which the users do not 
exploit their channel outputs for channel encoding ('restricted encoders'). We generalize 
Shannon's outer bound for two-way channels to correlated sources. 

Overall, our results characterize the necessary and sufficient conditions for reliable transmis- 
sion of correlated sources over various multiuser networks, hence answering question (1) for 
those scenarios. In these cases, the optimal performance is achieved by statistically independent 
source and channel codes (by either informational or operational separation), thus promising 
a level of modularity even when simply concatenating optimal source and channel codes is 
suboptimal. Hence, for the cases where we provide the optimal source-channel rate, we answer 
questions (2), (3) and (4) as well. 

The remainder of the paper is organized as follows. We review the prior work on joint source- 
channel coding for multiuser systems in Section HH and the notations and the technical tools that 
will be used throughout the paper in Section [Till In Section [IV] we introduce the system model 
and the definitions. The next four sections are dedicated to the analysis of special cases of the 
general system model. In particular, we consider multiple access channel model in Section |Vj 
compound multiple access channel model in Section |VT1 interference channel model in Section 
IVIII and finally the two-way channel model in Section IVIIII Our conclusions can be found in 
Section |IX] followed by the Appendix. 

II. Prior Work 

The existing literature provides limited answers to the four questions stated in Section U in 
specific settings. For the MAC with correlated sources, finite-letter sufficient conditions for 
achievability of a source-channel rate are given in [4] in an attempt to resolve the first problem; 
however, these conditions are later shown not to be necessary by Dueck [11]. The correlation 
preserving mapping technique of [4] used for achievability is later extended to source coding 
with side information via multiple access channels in [12], to broadcast channels with correlated 
sources in [13], and to interference channels in [14]. In [15], [16] a graph theoretic framework was 
used to achieve improved source-channel rates for transmitting correlated sources over multiple 
access and broadcast channels, respectively. A new data processing inequality was proved in [17] 
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that is used to derive new necessary conditions for reliable transmission of correlated sources 
over MACs. 

Various special classes of source-channel pairs have been studied in the literature in an effort 
to resolve the third question above, looking for the most general class of sources for which the 
comparison of the underlying source coding rate region and the capacity region is sufficient to 
determine the achievability of a source-channel rate. Optimality of separation in this classical 
sense is proved for a network of independent, non-interfering channels in [7]. A special class of 
the MAC, called the asymmetric MAC, in which one of the sources is available at both encoders, 
is considered in [8] and the classical source-channel separation optimality is shown to hold with 
or without causal perfect feedback at either or both of the transmitters. In [9], it is shown that for 
the class of MACs for which the capacity region cannot be enlarged by considering correlated 
channel inputs, classical separation is optimal. Note that all of these results hold for a special 
class of MACs and arbitrary source correlations. 

There have also been results for joint source-channel codes in broadcast channels. Specifically, 
in [6], Tuncel finds the optimal source-channel rate for broadcasting a common source to multiple 
receivers having access to different correlated side information sequences, thus answering the first 
question. This work also shows that the comparison of the broadcast channel capacity region and 
the minimum source coding rate region is not sufficient to decide whether reliable transmission 
is possible. Therefore, the classical informational source-channel separation, as stated in the 
third question, does not hold in this setup. Tuncel also answers the second and fourth questions, 
and suggests that we can achieve the optimal source-channel rate by source and channel codes 
that are statistically independent, and that, for the achievability of a source-channel rate b, the 
intersection of two regions, one solely depending on the source distributions, and a second one 
solely depending on the channel distributions, is necessary and sufficient. The codes proposed 
in [6] consist of a source encoder that does not use the correlated side information, and a joint 
source-channel decoder; hence they are not stand-alone source and channel codes^. Thus the 
techniques in [6] require the design of new codes appropriate for joint decoding with the side 

'Here we note that the joint source-channel decoder proposed by Tuncel in [6] can also be implemented by separate source 
and channel decoders in which the channel decoder is a list decoder [19] that outputs a list of possible channel inputs. However, 
by stand-alone source and channel codes, we mean unique decoders that produce a single codeword output, as it is understood 
in the classical source-channel separation theorem of Shannon. 
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information; however, it is shown in [18] that the same performance can be achieved by using 
separate source and channel codes with a specific message passing mechanism between the 
source/channel encoders/decoders. Therefore we can use existing near-optimal codes to achieve 
the theoretical bound. 

Broadcast channel in the presence of receiver message side information, i.e., messages at 
the transmitter known partially or totally at one of the receivers, is also studied from the 
perspective of achievable rate regions in [20] - [23]. The problem of broadcasting with receiver 
side information is also encountered in the two-way relay channel problem studied in [24], [25]. 

III. Preliminaries 

A. Notation 

In the rest of the paper we adopt the following notational conventions. Random variables will 
be denoted by capital letters while their realizations will be denoted by the respective lower 
case letters. The alphabet of a scalar random variable X will be denoted by the corresponding 
calligraphic letter X, and the alphabet of the n-length vectors over the n-fold Cartesian product 
by X n . The cardinality of set X will be denoted by \X\. The random vector (Xi, . . . , X n ) will be 
denoted by X n while the vector (X i: X i+1 , . . . ,X n ) by X™, and their realizations, respectively, 
by (xi, . . . , x n ) or x n and (x u x i+1 , ...,x n ) or xf. 

B. Types and Typical Sequences 

Here, we briefly review the notions of types and strong typicality that will be used in the 
paper. Given a distribution p x , the type P x n of an n-tuple x n is the empirical distribution 

P xn = -N(a\x n ) 
n 

where N(a\x n ) is the number of occurances of the letter a in x n . The set of all n-tuples x n 

with type Q is called the type class Q and denoted by Tq. The set of 5-strongly typical n-tuples 

according to Px is denoted by TmU and is defined by 

<5VaeX and N(a\x n ) = whenever P x (x) = 



T^ x]s = {xeX n ' 



-N(a\x n )-P x (a 
n 



The definitions of type and strong typicality can be extended to joint and conditional distri- 
butions in a similar manner [1]. The following results concerning typical sets will be used in 
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Fig. 1. The general system model for transmitting correlated sources over multiuser channels with correlated side information. 
In the MAC scenario, we have only one receiver Rxi; in the compound MAC scenario, we have two receivers which want to 
receive both sources, while in the interference channel scenario, we have two receivers, each of which wants to receive only its 
own source. The compound MAC model reduces to the "restricted" two-way channel model when W™ = SJ™ for i — 1,2. 



the sequel. We have 



< 



\X\ 



(1) 



-hg\T(k ]s \-H(X) 

for sufficiently large n. Given a joint distribution Pxy, if (xi,Ui) is drawn independent and 
identically distributed (i.i.d.) with PxPy for % = 1, • • • , n, where P x and P Y are the marginals, 



then 



Pr{{x n ,y n ) G T^ XYh } < 2~< I{ - X > Y ^ & \ 



(2) 



Finally, for a joint distribution Pxyz, if (xuVhZi) is drawn i.i.d. with PxPyPz for i = 
1, . . . , n, where Px, Py an d Pz are the marginals, then 

Pr{(x n ,y n ,z n ) G T( XYZ]g } < 2 ^ I ^Y,z) + ior;X,z)+i{z i Y^)-4S) _ (3) 

IV. System Model 

We introduce the most general system model here. Throughout the paper we consider various 
special cases, where the restrictions are stated explicitly for each case. 

We consider a network of two transmitters Txi and Tx 2 , and two receivers Rxi and Rx 2 . 
For i — 1,2, the transmitter Tx« observes the output of a discrete memoryless (DM) source Si, 
while the receiver Rx, observes DM side information Wt. We assume that the source and the 
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side information sequences, {Si^, S 2i k, Wi,fc, ^2,k}T=i are *■•*■•&• and are drawn according to a 
joint probability mass function (p.m.f.) p(s\, s 2 , wi, w 2 ) over a finite alphabet <Si x iS> 2 x VVi x W 2 . 
The transmitters and the receivers all know this joint p.m.f., but have no direct access to each 
other's information source or the side information. 

The transmitter Txj encodes its source vector S™ = (S^i, . . . , Si, m ) into a channel codeword 
X™ = (Xj :1 , . . . , X ijfl ) using the encoding function 

ft ,n) ■ $r - K n , (4) 

for i = 1,2. These codewords are transmitted over a DM channel to the receivers, each of which 
observes the output vector Y™ = (l^,i, . . . , Y^ n ). The input and output alphabets Xi and 3^i are all 
finite. The DM channel is characterized by the conditional distribution Py 1 ,y 2 |x 1 ,x 2 (yi) 2/2^1, ^2)- 
Each receiver is interested in one or both of the sources depending on the scenario. Let receiver 
Rxj form the estimates of the source vectors S™ and S™, denoted by S™\ and S™^, based on its 
received signal Y™ and the side information vector W™ = (W i: i, . . . , W i:m ) using the decoding 
function 

Due to the reliable transmission requirement, the reconstruction alphabets are the same as the 
source alphabets. In the MAC scenario, there is only one receiver Rx l5 which wants to receive 
both of the sources Si and S 2 . In the compound MAC scenario, both receivers want to receive 
both sources, while in the interference channel scenario, each receiver wants to receive only its 
own transmitter's source. The two-way channel scenario cannot be obtained as a special case of 
the above general model, as the received channel output at each user can be used to generate 
channel inputs. On the other hand, a "restricted" two-way channel model, in which the past 
channel outputs are only used for decoding, is a special case of the above compound channel 
model with W™ 1 = S™ for i — 1,2. Based on the decoding requirements, the error probability of 
the system, P e ( m > n ) will be defined separately for each model. Next, we define the source-channel 
rate of the system. 

Definition 4.1: We say that source-channel rate b is achievable if, for every e > 0, there exist 
positive integers m and n with n/m = b for which we have encoders f\ m ' n ' and f 2 , and 
decoders g { ™' n) and g { 2 m ' n) with decoder outputs (Sft, S™ 2 ) = g^Y?, W™), such that P^ < e. 
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V. Multiple Access Channel 

We first consider the multiple access channel, in which we are interested in the reconstruction 
at receiver Rxi only. For encoders /j and a decoder gj m , the probability of error for the 
MAC is defined as follows: 



p(m,n) a Pr{(S7',S£ l )^(S£ 1 ,S£ 2 )} 

(5™,s™)eS™x5 2 m 

Note that this model is more general than that of [4] as it considers the availability of correlated 
side information at the receiver [29]. We first generalize the achievability scheme of [4] to our 
model by using the correlation preserving mapping technique of [4], and limiting the source- 
channel rate b to 1. Extension to other rates is possible as in Theorem 4 of [4]. 

Theorem 5.1: Consider arbitrarily correlated sources Si and S 2 over the DM MAC with 
receiver side information W\. Source-channel rate b = 1 is achievable if 

H(S 1 \S 2 ,W 1 ) < I(X 1 ;Y 1 \X 2 ,S 2 ,W 1 ,Q), 

h(s 2 \s 1 ,w 1 ) < /(x 2; r 1 |x 1 ,5 1 ,^ 1 ,g), 

H(S 1 ,S 2 \U,W 1 ) < I(X 1 ,X 2 ;Y 1 \U,W 1 ,Q), 
and 

HiS^S^W^ < I(X 1 ,X 2 ;Y 1 \W l ), 
for some joint distribution 

p(q, si, s 2 , W!, x ± ,x 2 , yi) = p(q)p(s 1 , s 2 , wi)p(xi|g, s 1 )p(x 2 \q, s 2 )p(y 1 \x 1 , x 2 ) 

and 

U = f(S 1 ) = g(S 2 ) 

is the common part of Si and S 2 in the sense of Gacs and Korner [26]. We can bound the 
cardinality of Q by min{|Ai| • \X 2 \, \y\}. 

We do not give a proof here as it closely resembles the one in [4]. Note that correlation among 
the sources and the side information both condenses the left hand side of the above inequalities, 
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and enlarges their right hand side, compared to transmitting independent sources. While the 
reduction in entropies on the left hand side is due to Slepian-Wolf source coding, the increase in 
the right hand side is mainly due to the possibility of generating correlated channel codewords 
at the transmitters. Applying distributed source coding followed by MAC channel coding, while 
reducing the redundancy, would also lead to the loss of possible correlation among the channel 
codewords. However, when Si — W\ — S 2 form a Markov chain, that is, the two sources are 
independent given the side information at the receiver, the receiver already has access to the 
correlated part of the sources and it is not clear whether additional channel correlation would 
help. The following theorem suggests that channel correlation preservation is not necessary in 
this case and source-channel separation in the informational sense is optimal. 

Theorem 5.2: Consider transmission of arbitrarily correlated sources Si and S 2 over the DM 
MAC with receiver side information W\, for which the Markov relation S\ — W\ — Si holds. 
Informational separation is optimal for this setup, and the source-channel rate b is achievable if 

H{S 1 \W l ) < b-I(X i; Yi\X 2 ,Q), (6a) 

H(S 2 \Wi) < b-I(X 2 ;Yi\Xi,Q), (6b) 

and 

HfalWJ + HfalWi) < b-I(Xi,X 2 ;Yi\Q), (6c) 

for some joint distribution 

p(q,x 1 ,x 2 ,yi) = p(q)p(x 1 \q)p(x 2 \q)p(y 1 \x 1 ,x 2 ), (7) 

with \Q\ < 4. 

Conversely, if the source-channel rate b is achievable, then the inequalities in © hold with 
< replaced by < for some joint distribution of the form given in ©. 

Proof: We start with the proof of the direct part. We use Slepian-Wolf source coding 
followed by multiple access channel coding as the achievability scheme; however, the error 
probability analysis needs to be outlined carefully since for the rates within the rate region 
characterized by the right-hand side of © we can achieve arbitrarily small average error 
probability rather than the maximum error probability [1]. We briefly outline the code generation 
and encoding/decoding steps. 
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Consider a rate pair (R 1 ,R 2 ) satisfying 

F(^ 1 |W 1 )< i?! <6-/(X 1 ;F 1 |X 2 ,Q), (8a) 

H(S 2 \W 1 )< R 2 <b-I(X 2 ;Y 1 \X 1 ,Q), (8b) 

and 

H{S 1 \Wi) + H{S 2 \W!) < Ri + R 2 <b-I(X l ,X 2 ;Y 1 \Q). (8c) 

Code generation: At transmitter k, k = I, 2, independently assign every s™ G 5™ to one of 
the 2 mfifc bins with uniform distribution. Denote the bin index of s™ by ik(s™) e {1, . . . , 2 mRk }. 
This constitutes the Slepian-Wolf source code. 

Fix p(q), p(xi\q) and p(x 2 \q) such that the conditions in © are satisfied. Generate q n by 
choosing q, t independently from p(q) for i = 1, . . . , n. For each source bin index i^ — 1, . . . , 2 mRfe 
of transmitter k, k = 1,2, generate a channel codeword x%(i k ) by choosing x ki (i k ) independently 
from p(xfc|gj). This constitutes the MAC code. 

Encoders: We use the above separate source and the channel codes for encoding. The source 
encoder k finds the bin index of s™ using the Slepian-Wolf source code, and forwards it to the 
channel encoder. The channel encoder transmits the codeword x'1 corresponding to the source 
bin index using the MAC code. 

Decoder: We use separate source and channel decoders. Upon receiving y™, the channel 
decoder tries to find the indices (i[,i 2 ) such that the corresponding channel codewords satisfy 
(q n , #i(*i) , x 2 (i 2 )) E T?q XiX2 yi 5 - If one such pair is found, call it (i^i^). If no or more than 
one such pair is found, declare an error. 

Then these indices are provided to the source decoder. Source decoder tries to find s™ k such 
that ikis™) = i' k and (s£\ W™) G T^ kW i. If one such pair is found, it is declared as the output. 
Otherwise, an error is declared. 

Probability of error analysis: For brevity of the expressions, we define s = (s'i',s 2 n ), S = 
(S™, S™) and s = (s™ 1; s™ 2 ). The indices corresponding to the sources are denoted by i = 
{ii{s r { l ),i 2 {s 2 n )), and the indices estimated at the channel decoder are denoted by i' = (i'i,i 2 ). 
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The average probability of error can be written as follows: 
p(m,n) a £p{8^ 8 |s = s}p(s) 

s 

= J2 [p{s ^ s |i = i', S = s}p{i = i'|S = s) + P{s ^ s|i ^ i', S = s}p{i ^ i'|S = s)]p(s) 

s 

< £ [ p { § ^s|i = i / ,S = s} +p(i ^ i'|S = s)]p(s) 

s 

= ^P{Ms|i = i / ,S = s}p(s) + ^p(i^i / |S = sMs) (9) 

s s 

Now, in © the first summation is the average error probability given the fact that the receiver 
knows the indices correctly. This can be made arbitrarily small with increasing m, which follows 
from the Slepian-Wolf theorem. The second term in © is the average error probability for the 
indices averaged over all source pairs. This can also be written as 

$>(i ^ i'|S = s)p(s) = $>(i ^ i',I = i) 

s i 

= 5>(i^i'|I = i) P (I = i) 
i 

= ^^)X>(i^i'|I = i) (10) 

i 

where (flOl) follows from the uniform assignment of the bin indices in the creation of the source 



code. Note that (flOl) is the average error probability expression for the MAC code, and we know 
that it can also be made arbitrarily small with increasing m and n under the conditions of the 
theorem [1]. 

We note here that for 6=1 the direct part can also be obtained from Theorem 15.11 For this, 
we ignore the common part of the sources and choose the channel inputs independent of the 
source distributions, that is, we choose a joint distribution of the form 

p(q, s 1 , s 2 , w 1 ,x 1 , x 2 , Vi) = p{q)p{si, s 2 , w 1 )p{x 1 \q)p(x 2 \q)p{yi\x 1 , x 2 ). 

From the conditional independence of the sources given the receiver side information, both the 
left and the right hand sides of the conditions in Theorem 15. II can be simplified to the sufficiency 
conditions of Theorem 15.21 

We next prove the converse. We assume P e ( m,n ) — * for a sequence of encoders /j 
(i = 1,2) and decoders g( m ' n ) as n,m — >• oo with a fixed rate b = n/m. We will use Fano's 
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inequality, which states 

H{S?, S™\S^, S^ 2 ) < 1 + mP^ log |5i x S 2 \, 

= mS(P^ m ' n) ), (11) 

where 5(x) is a non-negative function that approaches zero as x — > 0. We also obtain 

tj I om om I om om \ \ tj I om\ om om \ /n\ 

n{b 1 ,b 2 \b 1A ,b 12 ) > tl\b l \b lx ,b l2 ), KYI) 

> H(S?\Y?,W™), (13) 

where the first inequality follows from the chain rule of entropy and the nonnegativity of the 
entropy function for discrete sources, and the second inequality follows from the data processing 
inequality. Then we have, for % = 1, 2, 

H{S™ \Y?, W™) < m<5(P e (m ' n) ). (14) 

We have 

-I(X?;YT\X2,Wr) > -I(S^]Yr-\Wr,X^), (15) 

n n 

= -[H{ST\W^, X") - H(S?\Y?, W?, X%)), (16) 

= -[HiS^W^-HiS^Y^W™^?)], (17) 

n 

> -\H(S?\Wr) - H(S?\Y"W?)l (18) 

n 



1 
> - 

- b 



H(S 1 \W 1 )-5(P^ m ' n) )\, (19) 

where tfB} follows from the Markov relation S? - X? - Y? given (X£, W^)\ (jnj) from the 



Markov relation X 2 — W™ — S™; f[18j) from the fact that conditioning reduces entropy; and (1191) 
from the memoryless source assumption and from (fTTI) which uses Fano's inequality. 
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On the other hand, we also have 



I(X?;Y?\X2,W? 



H{YT\X$, WD - H{Y?\X?, X%, WD, 

n 

HiYflXlWn-Y.HiY^Yr^X^XlW?), 



t=i 



= H{Y?\X»,W?) -^HiY^X^X^W™), 

i=l 

< J2H(Y ltt \X 2l ,Wn -J2 H ( Y i,i\ X u, X 2i,W? 
1=1 

■II 

= ^/(x H; r M |x 2j ,iyr 



8=1 



(20) 
(21) 

(22) 

(23) 

(24) 



i=l 



where (|2T|) follows from the chain rule; (1221) from the memoryless channel assumption; and ([23 
from the chain rule and the fact that conditioning reduces entropy. 

For the joint mutual information we can write the following set of inequalities: 



-I(X?,XbY?\W?) > -I(S^,S^;Y^\Wn, 
n n 

= -[H(S™, S™\WD - H(S™, S?\Y?, W?)], 



(25) 
(26) 



= -[HiSyiWT 1 ) + H(S™\W™) - H(S™, S™\Y?, W™)}, (27) 

n ,- 

> - \H(S? I WD + H(S? I WD - H(S?, S? I S?, S?), (28) 
n 



1 
> - 

- b 



H(S 1 \W 1 ) + H(S 2 \Wx) - 6{P ( 



(m,n)\ 



(29) 



where ([25]) follows from the Markov relation (S?, SI?) - {X? X 2 n ) - Y? given W?\ ([27]) from 
the Markov relation S? - W? - S?; tf28} from the fact that (S?, S?) - (Y/ 1 , W^) - (ST, S?) 
form a Markov chain; and ([29]) from the memoryless source assumption and from ([111) which 
uses Fano's inequality. 

By following similar arguments as in ([20l) - ([24]) above, we can also show that 

n 

I(X«,X»',Y?\W?) < ^/(X l4 ,X 24 ;y M |^r). (30) 

Now, we introduce a time-sharing random variable Q independent of all other random vari- 
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ables. We have Q = i with probability 1/n, i E {1,2,..., n}. Then we can write 

1 1 n 

-I{X?;Y?\X2,WT) < -Y,nXH;Y hl \X 2l ,W?), (31) 

= -J2l(X lq -Y,\X 2q ,Wr,Q = i), (32) 

1=1 

= I(X iQ ;Yq\X 2Q ,W™ 7 Q), (33) 

= I(X 1 ;Y\X 2 ,Q), (34) 

where X x = X lQ , X 2 = X 2Q , Y = Y Q , and Q = (W^,Q). Since S? and S 2 m , and therefore 
Xij and X 2i , are independent given W™, for g = (wj", i) we have 

Pr{X 1 = x 1 ,X 2 = a; 2 |g = g} = Pr{X u = x u X 2i = x 2 \W? = wj", Q = i} 

= Pr{X u = xxliy™ = <\<Q = «}Pr{X 2i = x 2 |W 1 m =w?,Q = i} 
= Pr{X 1 \Q = q}-Pr{X 2 \Q = q}. 

Hence, the probability distribution is of the form given in Theorem 15.21 
On combining the inequalities above we can obtain 

H(S 1 \W 1 )-S(P^) < bI{X i; Y\X 2 ,Q), (35) 

H{S 2 \ Wi) - 5(P^) < bI{X 2 ; Y\X U Q), (36) 

and 

H(S 1 \W 1 ) + H{3 2 \W!) - S(P^) < bI{Xi, X 2 - Y\Q). (37) 

Finally, taking the limit as m, n — ► oo and letting P e ( mn ) — »• leads to the conditions of the 
theorem. ■ 

To the best of our knowledge, this result constitutes the first example in which the underlying 
source structure leads to the optimality of (informational) source-channel separation independent 
of the channel. We can also interpret this result as follows: The side information provided to the 
receiver satisfies a special Markov chain condition, which enables the optimality of informational 
source-channel separation. We can also observe from Theorem 15.21 that the optimal source- 
channel rate in this setup is determined by identifying the smallest scaling factor b of the MAC 
capacity region such that the point (H(Si\Wi),H(S 2 ,Wi)) falls into the scaled region. This 
answers question (3) affirmatively in this setup. 
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A natural question to ask at this point is whether providing some side information to the 
receiver can break the optimality of source-channel separation in the case of independent mes- 
sages. In the next theorem, we show that this is not the case, and the optimality of informational 
separation continues to hold. 

Theorem 5.3: Consider independent sources S\ and S 2 to be transmitted over the DM MAC 
with correlated receiver side information W\. If the joint distribution satisfies p(si,s 2 , W\) = 
p(si)p(s 2 )p(wi\si, s 2 ), then the source-channel rate b is achievable if 

HiSxlS^Wt) < b-I(X 1 ;Y 1 \X 2 ,Q) 1 (38) 

H(S 2 \S 1 ,W 1 ) < b-IiXxY^XuQ), (39) 

and 

H(S l ,S 2 \W 1 ) < b-IiX^X^YtlQ), (40) 

for some input distribution 

p(q,xi,x 2 ,yi) = p(q)p(xi\q)p(x 2 \q)p(yi\xi,x 2 ), (41) 

with \Q\ < 4. 

Conversely, if the source-channel rate b is achievable, then d38T)- (|40l) hold with < replaced by 
< for some joint distribution of the form given in (BTj) . Informational separation is optimal for 
this setup. 

Proof: The proof is given in Appendix HI ■ 

Next, we illustrate the results of this section with some examples. Consider binary sources 
and side information, i.e., Si = S 2 = Wi = {1, 2}, with the following joint distribution: 

P Sl s 2Wl {Si = 0,S 2 = 0, W 1 = 0} = P Sl s 2 wASi = l,S 2 = 1, Wt = 1} = 1/3 

and 

Ps^wASi =0,S 2 = 1, Wi = 0} = Ps^wASi = 0,S 2 = 1, W x = 1} = 1/6. 

As the underlying multiple access channel, we consider a binary input adder channel, in which 
X X = X 2 = {0, 1}, y = {0, 1, 2} and 

Y = X 1 + X 2 . 
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H(S 2 \S!) 



0.5 
H(S 2 \W 1 ) 




H(Si\Wi) 0.5 H(S!\S 2 ) 



1.5 1.58 



Fig. 2. Capacity region of the binary adder MAC and the source coding rate regions in the example. 



Note that, when the side information W% is not available at the receiver, this model is the same 
as the example considered in [4], which was used to show the suboptimality of separate source 
and channel codes over the MAC. 

When the receiver does not have access to side information W\, we can identify the separate 
source and channel coding rate regions using the conditional entropies. These regions are shown 
in Fig. |2l The minimum source-channel rate is found as b = 1.58/1.5 = 1.05 cupss in the 
case of separate source and channel codes. On the other hand, it is easy to see that uncoded 
transmission is optimal in this setup which requires a source-channel rate of b = 1 cupss. Now, 
if we consider the availability of the side information W\ at the receiver, we have H(Si\Wi) = 
H(S2\Wi) = 0.46. In this case, using Theorem 15 .2[ the minimum required source-channel rate 
is found to be b = 0.92 cupss, which is lower than the one achieved by uncoded transmission. 

Theorem 15.31 states that, if the two sources are independent, informational source-channel 
separation is optimal even if the receiver has side information given which independence of the 
sources no longer holds. Consider, for example, the same binary adder channel in our example. 
We now consider two independent binary sources with uniform distribution, i.e., P(S\ = 0) = 
P(S 2 = 0) = 1/2. Assume that the side information at the receiver is now given by W 1 = 
X\ © X 2 , where © denotes the binary xor operation. For these sources and the channel, the 
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minimum source-channel rate without the side information at the receiver is found as b = 1.33 
cupss. When W\ is available at the receiver, the minimum required source-channel rate reduces 
to b = 0.67 cupss, which can still be achieved by separate source and channel coding. 

Next, we consider the case when the receiver side information is also provided to the trans- 
mitters. From the source coding perspective, i.e., when the underlying MAC is composed of 
orthogonal finite capacity links, it is known that having the side information at the transmitters 
would not help. However, it is not clear in general, from the source-channel rate perspective, 
whether providing the receiver side information to the transmitters would improve the perfor- 
mance. 

If Si — W\ — S 2 form a Markov chain, it is easy to see that the results in Theorem 15 . 2 1 continue 
to hold even when W\ is provided to the transmitters. Let Si = (Si,Wi) be the new sources 
for which Si — W\ — S 2 holds. Then, we have the same necessary and sufficient conditions as 
before, hence providing the receiver side information to the transmitters would not help in this 
setup. 

Now, let Si and Si be two independent binary random variables, and W\ — Si © Si. In this 
setup, providing the receiver side information W\ to the transmitters means that the transmitters 
can learn each other's source, and hence can fully cooperate to transmit both sources. In this 
case, source-channel rate b is achievable if 

H(S 1 ,S 2 \W 1 )<bI(X 1 ,X 2 ;Y 1 ) (42) 

for some input distribution p(xi, x 2 ), and if source-channel rate b is achievable then (1421) holds 
with < for some p(xi,Xi). On the other hand, if W\ is not available at the transmitters, we 
can find from Theorem 15.31 that the input distribution in (|42|) can only be p(xi)p(x 2 ). Thus, in 
this setup, providing receiver side information to the transmitters potentially leads to a smaller 
source-channel rate as this additional information may enable cooperation over the MAC, which 
is not possible without the side information. In our example of independent binary sources, the 
total transmission rate that can be achieved by total cooperation of the transmitters is 1.58 bits 
per channel use. Hence, the minimum source-channel rate that can be achieved when the side 
information W\ is available at both the transmitters and the receiver is found to be 0.63 cupss. 
This is lower than 0.67 cupps that can be achieved when the side information is only available 
at the receiver. 
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We conclude that, as opposed to the pure lossless source coding scenario, having side informa- 
tion at the transmitters might improve the achievable source-channel rate in multiuser systems. 

VI. Compound MAC with correlated sources 

Next, we consider a compound multiple access channel, in which two transmitters wish 
to transmit their correlated sources reliably to two receivers simultaneously [29]. The error 
probability of this system is given as follows: 



p(m,n) A p r ) (J (gin ? gm^ ^ ^m §r 



J k,2) 
fc=l,2 



E p(*r, *?)p U «i, C 2 ) * ( S T, s?) (st, st) = (s?, si 

The capacity region of the compound MAC is shown to be the intersection of the two MAC 
capacity regions in [27] in the case of independent sources and no receiver side information. 
However, necessary and sufficient conditions for lossless transmission in the case of correlated 
sources are not known in general. Note that, when there is side information at the receivers, 
finding the achievable source-channel rate for the compound MAC is not a simple extension 
of the capacity region in the case of independent sources. Due to different side information at 
the receivers, each transmitter should send a different part of its source to different receivers. 
Hence, in this case we can consider the compound MAC both as a combination of two MACs, 
and as a combination of two broadcast channels. We remark here that even in the case of single 
source broadcasting with receiver side information, informational separation is not optimal, but 
the optimal source-channel rate can be achieved by operational separation as is shown in [6]. 

We first state an achievability result for rate 6=1, which extends the achievability scheme 
proposed in [4] to the compound MAC with correlated side information. The extension to other 
rates is possible by considering blocks of sources and channels as superletters similar to Theorem 
4 in [4]. 

Theorem 6.1: Consider lossless transmission of arbitrarily correlated sources (Si,S 2 ) over a 
DM compound MAC with side information (Wi, W2) at the receivers as in Fig.Q] Source-channel 
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rate 1 is achievable if, for k = 1, 2, 

H{S 1 \S 2 ,W k ) < I(X i; Y k \X 2 ,S 2 ,W k ,Q), 
H(S 2 \S u W k ) < IiX^YklX^SuW^Q), 
H(S l7 S 2 \U,W k ) < I(X 1} X 2 ;Y k \U,W k ,Q), 

and 

H(S l7 S 2 \W k ) < I(X 1} X 2 ;Y k \W k ), 

for some joint distribution of the form 

p(q, Si, s 2 , w u w 2 , X!,x 2 , yi, y 2 ) = p{q)p{s u s 2 , w u w 2 )p{xi\q, Si)p(x 2 \q, s 2 )p(y l , y 2 \x u x 2 ) 

and 

U = /(5i) = g(S 2 ) 

is the common part of Si and S 2 in the sense of Gacs and Korner. 

Proof: The proof follows by using the correlation preserving mapping scheme of [4], and 
is thus omitted for the sake of brevity. ■ 

In the next theorem, we provide sufficient conditions for the achievability of a source-channel 
rate b. The achievability scheme is based on operational separation where the source and the 
channel codebooks are generated independently of each other. In particular, the typical source 
outputs are matched to the channel inputs without any explicit binning at the encoders. At the 
receiver, a joint source-channel decoder is used, which can be considered as a concatenation 
of a list decoder as the channel decoder, and a source decoder that searches among the list for 
the source codeword that is also jointly typical with the side information. However, there are 
no explicit source and channel codes that can be independently used either for compressing the 
sources or for independent data transmission over the underlying compound MAC. An alternative 
coding scheme composed of explicit source and channel coders that interact with each other is 
proposed in [18]. However, the channel code in this latter scheme is not the channel code for 
the underlying multiuser channel either. 

Theorem 6.2: Consider lossless transmission of arbitrarily correlated sources S\ and S 2 over 
a DM compound MAC with side information W\ and W 2 at the receivers. Source-channel rate 
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b is achievable if, for k = 1, 2, 

H{S 1 \S 2 ,W k ) < bI(X i; Y k \X 2 ,Q), (43) 

H(3 2 \S X , W k ) < bI(X 2 ; Y k \X h Q), (44) 

and 

H(S 1 ,S 2 \W k ) <bI(X 1 ,X 2 ;Y k \Q), (45) 

for some \Q\ < 4 and input distribution of the form p(q, Xi,x 2 ) = p(q) p{xi\q)p(x 2 \q). 

Remark 6.1: The achievability part of Theorem 16.21 can be obtained from the achievability 
of Theorem 16.11 Here, we constrain the channel input distributions to be independent of the 
source distributions as opposed to the conditional distribution used in Theorem 16.11 We provide 
the proof of the achievability of Theorem 16.21 below to illustrate the nature of the operational 
separation scheme that is used. 

Proof: Fix 5 k > and 7^ > for k — 1, 2, and Px 1 and P X2 . For b = n/m and k = 1, 2, 
at transmitter k, we generate M k = 2 m ^ H ^ Sk ^ +e ^ 2 ^ i.i.d. length-m source codewords and i.i.d. 
length-n channel codewords using probability distributions Ps k and Px k , respectively. These 
codewords are indexed and revealed to the receivers as well, and are denoted by s™(i) and x k (i) 
for 1 < i < M k . 

Encoder: Each source outcome is directly mapped to a channel codeword as follows: Given 
a source outcome S™ at transmitter m, we find the smallest i k such that S™ = s^(ifc), and 
transmit the codeword x^(i k ). An error occurs if no such i k is found at either of the transmitters 
fc = l,2. 

Decoder: At receiver k, we find the unique pair (i*,i 2 ) that simultaneously satisfies 

{x 1 (z 1 ),x 2 (i 2 ),Y k ) G ^[x 1 X 2 Y] Sk ^ 
and 

where T^ is the set of weakly 5-typical sequences. An error is declared if the [i\,i 2 ) pair is 
not uniquely determined. 
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Probability of error: We define the following events: 

£* = {(x^i),x n 2 ( i2 ),Y k n ) i T$ lX2Y]s J 
and 

EiUuh) = {(sTUiUTU*),™?) e ^^^ and (<(.?!), ^2), >?) G Tg^J 

Here, i?i denotes the error event in which either of the encoders fails to find a unique 
source codeword in its codebook that corresponds to its current source outcome. When such 
a codeword can be found, E k denotes the error event in which the sources S™ and S™ and the 
side information W^ at receiver k are not jointly typical. On the other hand, E k denotes the 
error event in which channel codewords that match the current source realizations are not jointly 
typical with the channel output at receiver k. Finally E k (ji,j 2 ) is the event that the source 
codewords corresponding to the indices j\ an d j 2 are jointly typical with the side information 
Wk and simultaneously that the channel codewords corresponding to the indices j\ an d j 2 are 
jointly typical with the channel output Y^. 

Define P fc (m ' n) 4 Pr{(S™, S 2 m ) ^ (%, %)}. Then P^ < £ fc=li2 P fe (m,n) . Again, from the 
union bound, we have 



pM<Pr{^} + Pr{^} + PrK}+ £ E k ( Ju j 2 ) + £ E k (j 1 J 2 ) + £ E k (j 1: j 2 ), 

(46) 



ji^u, ii=*i, ii^u. 

J2=12 J2^2 32^2 



where i\ and i 2 are the correct indices. We have 



E k 4 (ji,J2) = Pr{(s?(h),s™(j 2 ),Wn G T^ Wkhk }Pr{(xUji),x n 2 (j2),Y k n ) G rg^j } 



(47) 



In [6] it is shown that, for any A > and sufficiently large m, 

Pr{E k 1 } = (l-Pr{Sr = s k n (l)}) Mk 

_ 2 n[j-6A] 



exp"" 2 . (48) 
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We choose A < -^, and obtain Pr{i?i} — ► as m —* oo. 

Similarly, we can also prove that Px(Ei(k)) — > for i = 2,3 and fe = 1,2 as m,n -> oo 
using standard techniques. We can also obtain 

E Pr {(«rOU "SU), WT) G ^k^ } Pr {(*»&), $&), Y?) G T^^ } 

J2=«2 

< 2^[ H (Sl)+i)-m[I(Si;S2,W k )-X]-n[I(X 1 ;Y k \X 2 )-X\ ,4^ 

_ 2 -m[f-(b+l)A] ( 5 qj 

where in (|49b we used (0Q) and ©; and (l50l) holds if the conditions in the theorem hold. 

A similar bound can be found for the second summation in (|46b . For the third one, we have 
the following bound. 

E p r {Wii), *?&), WD G T^^ } Pr {(styi), ^2), *?) G T^^ } 

J27^2 

< 2m[^(5i)+e/2]+m[H(52)+e/2]2-m[/(5 1 ;52,VF fc )+/(52;5 1 ,W fc )-/(5 1 ;52|W fc )]-A] 2 -n[/(X 1 ,X 2 ;y fe )-A] ^^ 

< 2-m[H(5i|5 2 ,iy fc )+H(5 2 |5 1 ,iy fc )-W(X 1 ,X 2 ;y fc )-(b+l)A-e] 

= 2 _ m[e _ (b+ l )A] ^ ^ 2) 

where (|5D) follows from (OQ) and ©; and (l52l) holds if the conditions in the theorem hold. 

Choosing A < min I -^, 2(b+i) f » we can ma k e sure that all terms of the summation in (l46l) also 
vanish as m,n-^oo. Any rate pair in the convex hull can be achieved by time sharing, hence 
the time-sharing random variable Q. The cardinality bound on Q follows from the classical 
arguments. ■ 

We next prove that the conditions in Theorem 16.21 are also necessary to achieve a source- 
channel rate of b for some special settings, hence, answering question (2) affirmatively for these 
cases. We first consider the case in which Si is independent of (S 2 , Wi) and S2 is independent 
of (Si, W2) ■ This might model a scenario in which Rxi (Rx 2 ) and Tx 2 (Txi) are located close 
to each other, thus having correlated observations, while the two transmitters are far away from 
each other (see Fig. [3]). 

Theorem 6.3: Consider lossless transmission of arbitrarily correlated sources Si and S 2 over 
a DM compound MAC with side information W\ and W 2 , where Si is independent of (S 2 , Wi) 
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Fig. 3. Compound multiple access channel in which the transmitter 1 (2) and receiver 2(1) are located close to each other, and 
hence have correlated observations, independent of the other pair, i.e., Si is independent of (5*2, Wi) and S2 is independent of 
(Si,W 2 ) . 



and S 2 is independent of (Si,W 2 ) ■ Separation (in the operational sense) is optimal for this 
setup, and the source-channel rate b is achievable if, for (k, m) G {(1, 2), (2, 1)}, 



H{S k )<bI(X k ;Y k \X m ,Q), 
H(S m \W k ) <M(X m ;Y k \X k ,Q), 



and 



H(S k ) + H{S m \W k ) < bI(X k , X m , Y k \Q), 
for some | Q\ < 4 and input distribution of the form 

p(q,x 1 ,x 2 ) = p{q)p{xi\q)p{x 2 \q). 



(53) 

(54) 

(55) 
(56) 



Conversely, if source-channel rate b is achievable, then (I53l-(l55l) hold with < replaced by < 
for an input probability distribution of the form given in (l56l) . 

Proof: Achievability follows from Theorem l6.2[ and the converse proof is given in Appendix 

m ■ 

Next, we consider the case in which there is no multiple access interference at the receivers 
(see Fig. ID). We let Y k = (Y ltk , Y 2>k ) k = 1,2, where the memoryless channel is characterized 
by 



p{yi,i, 2/2,1, 2/1,2, 2/2,2^1, x 2 ) = p{yi,i, yi,2\xi)p{y 2 ,i,y 2 ,2\x2). 



(57) 



On the other hand, we allow arbitrary correlation among the sources and the side information. 
However, since there is no multiple access interference, using the source correlation to create 
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Fig. 4. Compound multiple access channel with correlated sources and correlated side information with no multiple access 
interference. 



correlated channel codewords does not enlarge the rate region of the channel. We also remark 
that this model is not equivalent to two independent broadcast channels with side information. 
The two encoders interact with each other through the correlation among their sources. 

Theorem 6.4: Consider lossless transmission of arbitrarily correlated sources S\ and S 2 over 
a DM compound MAC with no multiple access interference characterized by (1571) and receiver 
side information W\ and Wi (see Fig. |4]). Separation (in the operational sense) is optimal for 
this setup, and the source-channel rate b is achievable if, for (k,m) = {(1,2), (2, 1)} 



H(S k \S m ,W k ) < bI(X k ;Y k>k ), 
H(S m \S k ,W k ) < bI(X m ;Y m>k ), 



and 



H(S k ,S m \W k ) < b[I(X k ;Y k>k ) + I(X m ;Y m!k )}, 
for an input distribution of the form 

p(q,xi,x 2 ) = p(q)p(xi\q)p(x 2 \q). 



(58) 
(59) 

(60) 

(61) 



Conversely, if the source-channel rate b is achievable, then (|53l)-(l55l) hold with < replaced by 
< for an input probability distribution of the form given in (l56l) . 

Proof: The achievability follows from Theorem 16.21 by letting Q be constant and taking 
into consideration the characteristics of the channel, where (Xi, Y± fl , Y 1>2 ) is independent of 
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(X 2 ,Y 2>1 ,Y 2>2 ), The converse can be proven similarly to Theorem 16.31 and will be omitted for 
the sake of brevity. ■ 

Note that the model considered in Theorem 16.41 is a generalization of the model in [30] (which 
is a special case of the more general network studied in [7]) to more than one receiver. Theorem 
16.41 considers correlated receiver side information which can be incorporated into the model 
of [30] by considering an additional transmitter sending this side information over an infinite 
capacity link. In this case, using [30], we observe that informational source-channel separation 
is optimal. However, Theorem 16.41 argues that this is no longer true when the number of sink 
nodes is greater than one even when there is no receiver side information. 

The model in Theorem 16.41 is also considered in [31] in the special case of no side information 
at the receivers. In the achievability scheme of [31], transmitters first randomly bin their correlated 
sources, and then match the bins to channel codewords. Theorem 16.41 shows that we can achieve 
the same optimal performance without explicit binning even in the case of correlated receiver 
side information. 

In both Theorem 16.31 and Theorem 16.41 we provide the optimal source-channel matching 
conditions for lossless transmission. While general matching conditions are not known for 
compound MACs, the reason we are able to resolve the problem in these two cases is the 
lack of multiple access interference from users with correlated sources. In the first setup the 
two sources are independent, hence it is not possible to generate correlated channel inputs, 
while in the second setup, there is no multiple access interference, and thus there is no need 
to generate correlated channel inputs. We note here that the optimal source-channel rate in 
both cases is achieved by operational separation answering both question (2) and question (4) 
affirmatively. The supoptimality of informational separation in these models follows from [6], 
since the broadcast channel model studied in [6] is a special case of the compound MAC model 
we consider. We refer to the example provided in [31] for the suboptimality of informational 
separation for the setup of Theorem 16.41 even without side information at the receives. 

Finally, we consider the special case in which the two receivers share common side infor- 
mation, i.e., W\ = W 2 = W, in which case S\ — W — S 2 form a Markov chain. For example 
this models the scenario in which the two receivers are close to each other, hence they have the 
same side information. The following theorem proves the optimality of informational separation 
under these conditions. 
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Theorem 6.5: Consider lossless transmission of correlated sources S\ and S 2 over a DM 
compound MAC with common receiver side information W\ = W 2 = W satisfying Si — W — S 2 . 
Separation (in the informational sense) is optimal in this setup, and the source-channel rate b is 
achievable if, for k — 1 and 2, 

HfalW) < b-I(X i; Y k \X 2 ,Q), (62) 

H(S 2 \W) < b-I(X 2 ;Y k \X 1 ,Q), 

and 

H(S 1 \W) + H(S 2 \W) < b-I(X 1 ,X 2 ;Y k \Q), 

for some joint distribution p(q,X\,x 2 ,y) = p{q)p{x\\q) p(x 2 \q)p(y\xi, x 2 ), with \Q\ < 4. 

Conversely, if the source-channel rate b is achievable, then (l62l)- ((63l) hold with < replaced by 
< for an input probability distribution of the form given above. 

Proof: The achievability follows from informational source-channel separation, i.e, Slepian- 
Wolf compression conditioned on the receiver side information followed by an optimal compound 
MAC coding. The proof of the converse follows similarly to the proof of Theorem I5.2[ and is 
omitted for brevity. ■ 

VII. Interference channel with correlated sources 

In this section, we consider the interference channel (IC) with correlated sources and side 
information. In the IC each transmitter wishes to communicate only with its corresponding 
receiver, while the two simultaneous transmissions interfere with each other. Even when the 
sources and the side information are all independent, the capacity region of the IC is in general 
not known. The best achievable scheme is given in [32]. The capacity region can be characterized 
in the strong interference case [36], [10], where it coincides with the capacity region of the 
compound multiple access channel, i.e., it is optimal for the receivers to decode both messages. 
The interference channel has gained recent interest due to its practical value in cellular and 
cognitive radio systems. See [33] - [35] and references therein for recent results relating to the 
capacity region of various interference channel scenarios. 
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For encoders /j and decoders gf 1 , the probability of error for the interference channel 
is given as 

p (m,n) A p r I (J sn, ^ ^ 
[fe=l,2 

(s™,s™)e5™x<S^ [fc=l,2 

In the case of correlated sources and receiver side information, sufficient conditions for the 
compound MAC model given in Theorem 16.11 and Theorem 16.21 serve as sufficient conditions for 
the IC as well, since we can constrain both receivers to obtain lossless reconstruction of both 
sources. Our goal here is to characterize the conditions under which we can provide a converse 
and achieve either informational or operational separation similar to the results of Section [VH In 
order to extend the necessary conditions of Theorem 16.31 and Theorem 16.51 to ICs, we will define 
the 'strong source-channel interference' conditions. Note that the interference channel version 
of Theorem 16.41 is trivial since the two transmissions do not interfere with each other. 

The regular strong interference conditions given in [36] correspond to the case in which, for all 
input distributions at transmitter Tx 1; the rate of information flow to receiver Rx 2 is higher than 
the information flow to the intended receiver Rx x . A similar condition holds for transmitter Tx 2 
as well. Hence there is no rate loss if both receivers decode the messages of both transmitters. 
Consequently, under strong interference conditions, the capacity region of the IC is equivalent to 
the capacity region of the compound MAC. However, in the joint source-channel coding scenario, 
the receivers have access to correlated side information. Thus while calculating the total rate 
of information flow to a particular receiver, we should not only consider the information flow 
through the channel, but also the mutual information that already exists between the source and 
the receiver side information. 

We first focus on the scenario of Theorem 16.31 in which the source S\ is independent of 
(S 2 ,W 1 ) and S 2 is independent of (5*2, W-y). 

Definition 7.1: For the interference channel in which S\ is independent of (S 2 , W\) and S 2 
is independent of (S 2 , W\), we say that the strong source-channel interference conditions are 
satisfied for a source-channel rate b if, 

b • J(X i; Y!|X 2 ) < b ■ I(X i; Y 2 \X 2 ) + /(Si; W 2 ), (63) 
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and 

b ■ I(X 2 ; Y 2 \X X ) < b ■ I(X 2 ; Y 1 \X 1 ) + I(S 2 ; W x ), (64) 

for all distributions of the form p(wi,w 2 , Si, s 2 , x x ,x 2 ) — p(wi,w 2 , si, s 2 )p(xi\si)p(x 2 \s 2 ). 

For an IC satisfying these conditions, we next prove the following theorem. 

Theorem 7.1: Consider lossless transmission of Si and S 2 over a DM IC with side information 
W\ and W 2 , where Si is independent of (5*2, W\) and S 2 is independent of (S 2 , Wi). Assuming 
that the strong source-channel interference conditions of Definition 17.11 are satisfied for b, 
separation (in the informational sense) is optimal. The source-channel rate b is achievable if, the 
conditions (|43l) -(l45l) in Theorem 16.21 hold. Conversely, if rate b is achievable, then the conditions 
in Theorem 16.21 hold with < replaced by <. 

Before we proceed with the proof of the theorem, we first prove the following lemma. 

Lemma 7.2: If (Si, W 2 ) is independent of (S 2 , W\) and the strong source-channel interference 
conditions (I63l)-(l64l) hold, then we have 



I(X»;Y»\X?) < I(X2;Y?\X?)+I(S?;W™), (65) 

and 

I(XbY?\X«) < I(X?;Y 2 n \X2) + I(S™;W™) } (66) 

for all m and n satisfying n/m = b. 

Proof: To prove the lemma, we follow the techniques in [10]. Condition (l64l) implies 



I(X 2 ; Y 2 \Xi, U) - I(X 2 ; Yi\X u U) < l -I{S 2 - Wi) (67) 

for all U satisfying U - {X h X 2 ) - (Y h Y 2 ). 
Then as in [10], we can obtain 

1 l A 2 i Y 2 l A l ) ~ L l A 2 ' Y l l A l ) — 1 l A 2n, *2n| A l , Y 2 ) ~ J- l A 2n, Y ln\ A l , Y 2 ) 

_J_ T ( yn-1 . -yn-U yn y \ j( ytl-l, yn-l 1 yn -y \ 

+ -H A 2 ) Y 2 l A l J Y ln) — A A 2 ■> Y l \^li Y ln) 

=I{X 2n ; Y 2n \Xi n ) — I(X 2n ; Yi n \Xi n ) 

i T( vn— 1. vn— 1 1 vn— 1\ r/ vi-1. v-1 1 vn-l\ 
+ -H A 2 1 Y 2 l A l ) ~ 1 { Ji 2 J Y l l A l ) 

n 

= 2_J\i{x 2 i]Y 2i \Xii) - i{x 2 i\ Yu\Xii)\. 
t=i 
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Using the hypothesis (1641) of the theorem, we obtain 

71 

J(X"; Y?\X?) - I(X%; Y t n \X?) < -I(S 2 ; W 1 ) 

= I(S™;W™). 

Eqn. (l66l) follows similarly. ■ 

Proof: (of Theorem 17.71) Achievability follows by having each receiver decode both Si and 
S 2 , and then using Theorem 16. 1 1 We next prove the converse. From (l95l)-(l98l). we have 



-/(If; Y?\X$) > \ [H{S{) - 5(p( m ' n) )] . (68) 



n 
We can also obtain 

-/(X™; F 2 n |X 2 n ) > -[/(Xf; Y?\X%) - I{S™\ W™)}, (69) 

= i[fT(5'i) - 5(P e (m ' n) )] - ^I(ST; W™), (70) 

= l l \H{S 1 \W 2 )-5{PJT' n) )l (VI) 

in which d69]) follows from (|66|). and (J70l) from (|68"1) . 
Finally for the joint mutual information, we have 

i/(X{\ X 2 " ; Yi") =^[/(*r; l?) + H^; Y?\X?)l 

>-[I(S™; *?) + /(X 2 n ; F"|X") - I(S™; W7 1 )}, (72) 

>i[l(ST; 1?) + I(S?; Y 2 n \x?) - lis?; WT)], (73) 

=-[H(S?) - H{S™\Y?) + H{S?\X?) - H(S™\Y 2 n ,X?) 

lb 

+ H{S 2 n \W™)-H{S™)}, 
>-[H{S?) - H(S?\Y?) - H{S™\Y 2 n ) + H(S%\W?)\, (74) 

=-[if (ST) - H(S?\Y", W7 1 ) - H{S™\Y 2 n , W™) + H(S?\Wr)l (75) 
n 

>i[tf (Si) + # (S 2 |Wi) - 25(P e (m '" } )], (76) 



for any e > and large enough m and n, where ( 172)) follows from the data processing 
inequality and (1651) ; (T73j) follows from the data processing inequality since S™ — X 2 — Y 2 



form a Markov chain given Xf ; (|74|) follows from the independence of X" and S 1 ™ and the fact 
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Fig. 5. The two-way channel model with correlated sources. 



that conditioning reduces entropy; ( 175)) follows from the fact that S\ is independent of (£2, W\) 
and S 2 is independent of (S2, Wi); and (176)) follows from Fano's inequality. The rest of the 
proof closely resembles that of Theorem 16.31 ■ 

Next, we consider the IC version of the case in Theorem 16.51 in which the two receivers 
have access to the same side information W and with this side information the sources are 
independent. While we still have correlation between the sources and the common receiver side 
information, the amount of mutual information arising from this correlation is equivalent at both 
receivers since W\ = W 2 . This suggests that the usual strong interference channel conditions 
suffice to obtain the converse result. We have the following theorem for this case. 

Theorem 7.3: Consider lossless transmission of correlated sources Si and S 2 over the strong 
IC with common receiver side information W\ = W 2 = W satisfying Si — W — S 2 . Separation 
(in the informational sense) is optimal in this setup, and the source-channel rate b is achievable 
if and only if the conditions in Theorem 16.51 hold. 

Proof: The proof follows from arguments similar to those in the proof of Theorem 16.51 and 
results in [28], where we incorporate the strong interference conditions. ■ 



VIII. Two-way channel with correlated sources 

In this section, we consider the two-way channel scenario with correlated source sequences 
(see Fig. [5]). The two-way channel model was introduced by Shannon [3] who gave inner and 
outer bounds on the capacity region. Shannon showed that his inner bound is indeed the capacity 
region of the "restricted" two-way channel, in which the channel inputs of the users depend only 
on the messages (not on the previous channel outputs). Several improved outer bounds are given 
in [37]-[39] using the "dependence-balance bounds" proposed by Hekstra and Willems. 
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In [3] Shannon also considered the case of correlated sources, and showed by an example that 
by exploiting the correlation structure of the sources we might achieve rate pairs given by the 
outer bound. Here we consider arbitrarily correlated sources and provide an achievability result 
using the coding scheme for the compound MAC model in Section |VU It is possible to extend 
the results to the scenario where each user also has side information correlated with the sources. 

In the general two-way channel model, the encoders observe the past channel outputs and 
hence they can use these observations for encoding future channel input symbols. The encoding 
function at user i at time instant j is given by 

fij : S? x yr 1 - Xi, (W) 

for i — 1, 2, The probability of error for the two-way channel is given as 

p(m,n) A p r \ \J grn ^ §rn 
[fc=l,2 

E p(*r> s 2 1 )p \ U ^ ± *?|(sr, s™) = (s?, s -) 

Note that, if we only consider restricted encoders at the users, than the system model is equivalent 
to the compound MAC model of Fig. Q] with W^ = S x m and W^ = Sf. From Theorem O we 
obtain the following corollary. 

Corollary 8.1: In lossless transmission of arbitrarily correlated sources (Si, S 2 ) over a DM 
two-way channel, the source-channel rate b = 1 is achievable if 

H{S!\S 2 ) < I(X 1 ;Y 2 \X 2 ,S 2 ,Q) and 

H&lSt) < I(X 2 ;Y 1 \X 1 ,S 1 ,Q), 

for some joint distribution of the form 

p(q, si, s 2 , xi, x 2 , |/!, y 2 ) = p(q)p(si, s 2 )p(xi\q, s 1 )p(x 2 \q, s 2 )p(y u y 2 \xt, x 2 ). 
Note that here we use the source correlation rather than the correlation that can be created 
through the inherent feedback available in the two-way channel. This correlation among the 
channel codewords potentially helps us achieve source-channel rates that cannot be achieved 
by independent inputs. Shannon's outer bound can also be extended to the case of correlated 
sources to obtain a lower bound on the achievable source-channel rate as follows. 
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Proposition 8.2: In lossless transmission of arbitrarily correlated sources (Si, 5*2) over a DM 
two-way channel, if the source-channel rate b is achievable, then 

i^SilSz) < bI(X i; Y 2 \X 2 ) and 

HiSzlSj < bI(XxY x \X x ), 

for some joint distribution of the form 

p(s l7 s 2 , xi, x 2 , yi, 2/2) = p(si, s 2 )p(xi, x 2 )p(yi,y 2 \x!, x 2 ). 
Proof: We have 

H{S™ I S 2 m ) = /(S^; K,™ I S™) + if (5f I S™, y 2 n ) (78) 

< 1(5™; Y 2 n I 5 2 m ) + m5{p( m ' n) ) (79) 
= ff(lTISr) - H(Y 2 n \S™, 5 2 m ) + m5(P e (m ' n) ) (80) 

n 

= J2 H(Y 2i }S?, KT 1 ) - H(Y 2l \ST, 5 2 m , KT 1 ) + m5(P^) (81) 

j=i 

n 

< 53 if (F 2i |s 2 m , rr\ x») - # (y 2i |sr, ^ m , >r\ ^r 1 , x 2i ) + ms(p^) m) 
i=i 

n 

< 53 #(**!**) - H(Y 2i \S?, S™ Kp 1 , Fr 1 , X M , X 2i ) + m5(P e (m,n) ) (83) 

i=l 

n 

<Y,H{Y 2l \X 2i ) - H(Y 2l \X ll ,X 2l )+m5(P^) (84) 

j=i 

n 

< ^ I(X U ; y 2i |X 2l ) + m<KP e (m ' n) ) (85) 



j=i 



where (1791) follows from Fano's inequality; (|82l) follows since X 2 is a deterministic function 
of (S 2 \Y 2 l ~ l ) and the fact that conditioning reduces entropy; (|83l) follows similarly as Xf is 
a deterministic function of (S]™, Y"/ -1 ) and the fact that conditioning reduces entropy; and (f8~4l) 
follows since Y 2i - (X li} X 2i ) - (5™, 5 2 m , Kp 1 , Y? _1 ) form a Markov chain. 
Similarly, we can show that 

n 

H{S™\S?) < ^ I(X 2i ; Y ll \X ll ) + m8(Pt> n) ). (86) 

i=i 

From convexity arguments and letting m, n — > 00, we obtain 

^(Si|5 2 )<W(X i; y 2 |X 2 ), (87) 

F^ISO^fe/^;^^), (88) 
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for some joint distribution p(xi, x 2 ). ■ 

Remark 8.1: Note that the lower bound of Proposition l8.2l allows all possible joint distributions 
for the channel inputs. This lets us express the lower bound in a separable form, since the source 
correlation becomes useless to introduce any additional structure to the transmitted channel 
codewords. In general, not all joint channel input distributions can be achieved at the two users, 
and tighter bounds can be obtained by limiting the set of possible joint distributions as in [37]- 
[39]. 

However, if the existing source correlation allows the users to generate the optimal joint 
channel input distribution, then the achievable region given in Corollary 18.11 might meet the 
upper bound without the need to exploit the feedback to generate further correlation. This has 
been illustrated by an example in [3]. Shannon considered correlated binary sources Si and S 2 
such that 

P Sl s 2 (Si = 0, S 2 = 1) = P Sl s 2 (Si = 1, S 2 = 0) = 0.275 

and 

P Sl s 2 (Si = 1,5-2 = 1) = 0.45, 

and a binary multiplier two-way channel, in which 

Xi = x 2 = y 1 = y 2 = {o, 1} 

and 

Yi = Y 2 = Xi ■ X 2 . 



Using Proposition 18.21 we can set a lower bound of b = 1 on the achievable source-channel rate. 
On the other hand, the source-channel rate of 1 can be achieved simply by uncoded transmission. 
Hence, in this example, the correlated source structure enables the transmitter to achieve the 
optimal joint distribution for the channel inputs without exploiting the inherent feedback in the 
two-way channel. Note that the Shannon outer bound is not achievable in the case of independent 
sources in a binary multiplier two-way channel [37], and the achievable rates can be improved 
by using channel inputs dependent on the previous channel outputs. 
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IX. Conclusions 

We have considered source and channel coding over multiuser channels with correlated receiver 
side information. Due to the lack of a general source-channel separation theorem for multiuser 
channels, optimal performance in general requires joint source-channel coding. Given the dif- 
ficulty of finding the optimal source-channel rate in a general setting, we have analyzed some 
fundamental building-blocks of the general setting in terms of separation optimality. Specifically, 
we have characterized the necessary and sufficient conditions for lossless transmission over 
various fundamental multiuser channels, such as multiple access, compound multiple access, 
interference and two-way channels for certain source-channel distributions and structures. In 
particular, we have considered transmitting correlated sources over the MAC with receiver side 
information given which the sources are independent, and transmitting independent sources over 
the MAC with receiver side information given which the sources are correlated. For the compound 
MAC, we have provided an achievability result, which has been shown to be tight i) when each 
source is independent of the other source and one of the side information sequences, ii) when 
the sources and the side information are arbitrarily correlated but there is no multiple access 
interference at the receivers, iii) when the sources are correlated and the receivers have access to 
the same side information given which the two sources are independent. We have then showed 
that for cases (i) and (iii), the conditions provided for the compound MAC are also necessary 
for interference channels under some strong source-channel conditions. We have also provided 
a lower bound on the achievable source-channel rate for the two-way channel. 

For the cases analyzed in this paper, we have proven the optimality of designing source and 
channel codes that are statistically independent of each other, hence resulting in a modular system 
design without losing the end-to-end optimality. We have shown that, in some scenarios, this 
modularity can be different from the classical Shannon type separation, called the 'informational 
separation', in which comparison of the source coding rate region and the channel capacity 
region provides the necessary and sufficient conditions for the achievability of a source-channel 
rate. In other words, informational separation requires the separate codes used at the source 
and the channel coders to be the optimal source and the channel codes, respectively, for the 
underlying model. However, following [6], we have shown here for a number of multiuser 
systems that a more general notion of 'operational separation' can hold even in cases for which 
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informational separation fails to achieve the optimal source-channel rate. Operational separation 
requires statistically independent source and channel codes which are not necessarily the optimal 
codes for the underlying sources or the channel. In the case of operational separation, comparison 
of two rate regions (not necessarily the compression rate and the capacity regions) that depend 
only on the source and channel distributions, respectively, provides the necessary and sufficient 
conditions for lossless transmission of the sources. These results help us to obtain insights into 
source and channel coding for larger multiuser networks, and potentially would lead to improved 
design principles for practical implementations. 

Appendix I 
Proof of Theorem 15.31 

Proof: The achievability again follows from separate source and channel coding. We first 
use Slepian-Wolf compression of the sources conditioned on the receiver side information, then 
transmit the compressed messages using an optimal multiple access channel code. 

An alternative approach for the achievability is possible by considering W\ as the output of a 
parallel channel from Si, S2 to the receiver. Note that this parallel channel is used m times for 
n uses of the main channel. The achievable rates are then obtained following the arguments for 
the standard MAC: 

mH(Si) < I{S™,X?-Y{ l ,Wl n \X2,S™,Q) (89) 

= J(5r;^r|5 2 m ) + J(X 1 n ;y i n |X™,g) (90) 

= mI(Si;Wi\S 2 )+nI(Xi;Yi\X 2 ,Q), (91) 



and using the fact that p(si, s 2 ,Wi) = p(si)p(s2)p(wi\si, s 2 ) we obtain (1381) (similarly for (1391 
and (|40l) ). Note that, this approach provides achievable source-channel rates for general joint 
distributions of Si, S 2 and W\. 
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For the converse, we use Fano's inequality given in (fTTI) and (fl4"l) . We have 

h(X^Y?\X«) > ±I(S?;Y?\X»), (92) 

n n 

= ±I(S?,WT;Y?\X2), (93) 

n 

> -I(S?;Y?\X? ,!&?), 

n 

> ~[H(S?\S?,WT)-m5{Pl m ' n) )], (94) 



where ()92|) follows from the Markov relation S*] 71 — X™ — Y™ given X 2 ; f|93|) from the Markov 



relation Wf - (X£, S?) - Y?\ and ([94]) from Fano's inequality (fl4) . 
We also have 

1 n 1 



Similarly, we have 



1 n 1 

-£/(**; I^Xm) > -[H(S 2 \S l ,W 1 ) - S(P^)], 



and 

i n i 

-X)/(^it,^2i;n,i) > t^^i,^!^)-^™-"))]. 

n£J b 

As usual, we let p( m > n > — > 0, and introduce the time sharing random variable Q uniformly 
distributed over {1,2,..., n} and independent of all the other random variables. Then we define 
X 1 ± X 1Q , X 2 ± X 2Q and Y 1 ± Y 1Q . Note that Pr{X 1 = Xl ,X 2 = x 2 \Q = q} = Pr{X x \Q = 
q] ■ Pr{X 2 \Q = q] since the two sources, and hence the channel codewords, are independent 
of each other conditioned on Q. Thus, we obtain (l38l)- (|40l) for a joint distribution of the form 
(SB. ■ 
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We have 



Appendix II 
Proof of Theorem 16.31 



n " n 

=-[ij(5rix 2 n ) - h{st\y?,x2) 

>~[H{S?) - H(S?\Y?)], 

lb 



>- 

~b 



H(Sx) - 5{p( m ' n 



,n)\ 



(95) 
(96) 
(97) 
(98) 



for any e > and sufficiently large m and n, where (|95|) follows from the conditional data 
processing inequality since S™ — X™ — Y™ forms a Markov chain given X% ', (J97|) from the 
independence of S 1 ™ and X^ and the fact that conditioning reduces entropy; and ( |98|) from the 
memoryless source assumption, and from Fano's inequality. 

For the joint mutual information, we can write the following set of inequalities: 



-I{X^ XI- Y?) > h(Sr,S?;Yf), 
n n 

= -I(SY l ,S^,wr % ;Y^), 

n 

> -I(S?,S?-,Y?\W?), 

n 

1 



> 



[H(S?, S™\W™) - H(S™, S?\Y?, W™)}, 
[H(S?) + H{S r 2 n \W™) - H{S™, S™\Y{ 1 , W?)], 
H(Sx) + H(S 2 \W 1 ) - 5{P^ n) ) 



(99) 
(100) 
(101) 

(102) 
(103) 



for any e > and sufficiently large m and n, where (|99|) follows from the data processing 
inequality since (S?,S!?) - (X?,X$) - Y x n form a Markov chain; (flOO]) from the Markov 
relation W™ — (S™, S™) — Y"™; (UOip from the chain rule and the non-negativity of the mutual 
information; ( 1102J) from the independence of S 1 ™ and (S™, W™); and (1103J) from the memoryless 
source assumption and Fano's inequality. 
It is also possible to show that 



J2l(X u ;Y lt \X 2t ) > IiX^-^lX. 



2)i 



(104) 



i=l 
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and similarly for other mutual information terms. Then, using the above set of inequalities and 

letting Pj m < n ) -> 0, we obtain 

1 1 n 

rH(S 1 ) < -J2HXii;Yu\X 2i ), 

o n i=1 

1 1 n 

rH{s 2 \w x ) < -J2HX2i;Yu\Xii), 

b nfri 

and 

1 1 n 

-(H(S l ) + H(S 2 \W 1 )) < -X;i(X u ,X 2i ;Y u ), 

for any product distribution on X\ x X 2 . We can write similar expressions for the second receiver 
as well. Then the necessity of the conditions of Theorem 16.21 can be argued simply by inserting 
the time- sharing random variable Q. 
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